It's not only contributing back as it happens with code. We also want to increase popularity of MLonCode and attract more people. A dataset is always the starting point of any DS research. No data => no research.

What is meant by "reusable", it is not a typical concept. I would say that what makes a good dataset is:

1. discoverable
2. in-depth documented (this is different from (1)). So many times I saw cool data very poorly described. Also would be nice to see problem suggestions for newbies.
3. accessibility. Data should be indexed if it makes sense, also the format matters. As I heard from Konstantin the other day, "just give me the darn CSVs, I am tired of fighting with Siva". If the dataset is big and targets Spark, it should not be a single gzipped txt, it should be splittable lzo chunks + a tool for reading those files without Spark. Another example: data distributed through BTSync - nobody will serve it, it is not another hot pirated movie or a cracked AAA game.
4. reproducible. Few people really want it, but still.

"kernel" is not a typical word. It may be used in NN contexts, but in general, it is confusing. I would replace it with a "training algorithm". Usually, it trains the model, not generates it. The same thing with "inferencer", I would replace it with "application".

Awesome chart, I like it a lot!

This oxford comma really confuses me. I was about to propose changing "let" to "lets" but then realized it was about both things.

So this is the thing: there is a huge difference between a research paper and what we want to achieve. Papers are always limited in size and the authors desperately try to squeeze as much information as possible. This often leads to excluding important descriptions, which are not strictly necessary but simplify the reproduction.

Think of it as a physical experiment. A paper includes: initial conditions; methodology; observations throughout the experiment lifetime; the empirical results; explanations and conclusions. Unfortunately, not ML papers.

So a dream model documentation should have:

1. Thorough dataset documentation
2. Problem statement. This includes choosing the right quality metric.
3. Architecture description, metaparameter values. Random seed - this guy is so hard to get right and requires using exactly the same code.
4. All possible plots how the model was trained. Time observations.
5. Stability and deviations: how metaparameters influence, how stable is the training. E.g. different random seeds may lead to quite different results in case the model is buggy. Another example: we reduce the number of neurons 100x and get 1% accuracy drop: it is a fair tradeoff for many people.
6. Results: achieved metrics, examples.

Some of the issues are to be resolved by attaching the paper.

Yes, but we can't assume having a paper for everything. It won't be feasible from a time perspective.

Absolutely. I meant solely PGA here.

There is versioning, though not reflected in src-d/models. Models can derive, either with the relation to parent or not. E.g. it is a common situation when our data engineering and filtering are not perfect and we miss data or pass in garbage. In that case, the relation to the previous model is saved. Sometimes it is just a regular update without the relation to the previous one.

I think the parent relation, makes @campoy's analogy to containers even strong here. There is a lot we can/should learn from how Docker registry tackled this.

Another important notice: models contain dependencies to upstream models which were used in the generation process. Datasets are also models in this terminology and should have a UUID (yeah, this is confusing, I know).

The only way which I see to reference the code is to record the whole Python package dependency tree. This still misses the actual custom calling code in many cases, and I need to apply some dark Python alchemy to discover and record it. We also need to store it somewhere in the model file.

Since each model references the code it was created from. The dependency tree is there already.

We will die. Seriously. I tried it and it is completely out of maintenance. Special pain belongs to adding new models and being blocked for a few days until the repository is created. I am strongly against this idea.

There is also the central registry of our models in src-d/models which is of strong necessity as the only way to fetch the index and automatically download models in downstream apps.

We already have 5 models to date, and the only reason why it is so few is that we did not have data. Now that we have PGA, we will bake new models like pies, with tens of different architectures and problems. Models are not code repositories, there is no point in contributing to existing ones, it is always about adding smth new.

Besides, we need to solve the problem with the community, because we want to allow external people to push models into our registry. Think of it as DockerHub for models.

Think of it as DockerHub for models.

What I believe @campoy is proposing is DockerHub for models, datasets, training algorithms, applications etc.

Model is an artifact of a training algorithm. Algorithm is code and we can improve it, fix it, etc. So the algorithms should be on GitHub/Git, separate from the model storage.

This will not work. All the metadata should be generated automatically from the self-contained ASDF files. Otherwise this will be a nightmare to support.

It might seem obvious, but we should also include:

• date the dataset was generated
• date of retrieved data if applicable (this might be included in the link to the original data sources)

Do we want to standarize dataset version to something? Maybe just date, or semver?

semver sounds pretty good, but it's a solution so I don't wanna decide on it just yet
I was considering using Docker hub, which would bring this for free.